{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem\n",
    "The number of shoes sold by an e-commerce company during the first three months(12 weeks) of the year were:\n",
    "<br>\n",
    "23 21 19 24 35 17 18 24 33 27 21 23\n",
    "\n",
    "Meanwhile, the company developed some dynamic price optimization algorithms and the sales for the next 12 weeks were:\n",
    "<br>\n",
    "31 28 19 24 32 27 16 41 23 32 29 33\n",
    "\n",
    "Did the dynamic price optimization algorithm deliver superior results? Can it be trusted?\n",
    "\n",
    "### Solution\n",
    "\n",
    "Before we get onto different approaches, let's quickly get a feel for the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "sns.set(color_codes=True)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load the data\n",
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23.75"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "before_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27.916666666666668"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "after_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "observed_difference = after_opt.mean() - before_opt.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference between the means is: 4.16666666667\n"
     ]
    }
   ],
   "source": [
    "print \"Difference between the means is:\", observed_difference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, the sales after optimization is more than the sales before optimization. But is the difference legit? Could it be due to chance?\n",
    "\n",
    "**Classical Method** : We could cover this method later on. This entails doing a *t-test* \n",
    "\n",
    "**Hacker's Method** : Let's see if we can provide a hacker's perspective to this problem, similar to what we did in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Step 1: Create the dataset. Let's give Label 0 to before_opt and Label 1 to after_opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Learn about the following three functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.append"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?np.ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),\n",
    "np.append(before_opt, after_opt)], dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (2, 24)\n",
      "Data: \n",
      "[[ 0  0  0  0  0  0  0  0  0  0  0  0  1  1  1  1  1  1  1  1  1  1  1  1]\n",
      " [23 21 19 24 35 17 18 24 33 27 21 23 31 28 19 24 32 27 16 41 23 32 29 33]]\n"
     ]
    }
   ],
   "source": [
    "print \"Shape:\", shoe_sales.shape\n",
    "print \"Data:\", \"\\n\", shoe_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (24, 2)\n",
      "Data: \n",
      "[[ 0 23]\n",
      " [ 0 21]\n",
      " [ 0 19]\n",
      " [ 0 24]\n",
      " [ 0 35]\n",
      " [ 0 17]\n",
      " [ 0 18]\n",
      " [ 0 24]\n",
      " [ 0 33]\n",
      " [ 0 27]\n",
      " [ 0 21]\n",
      " [ 0 23]\n",
      " [ 1 31]\n",
      " [ 1 28]\n",
      " [ 1 19]\n",
      " [ 1 24]\n",
      " [ 1 32]\n",
      " [ 1 27]\n",
      " [ 1 16]\n",
      " [ 1 41]\n",
      " [ 1 23]\n",
      " [ 1 32]\n",
      " [ 1 29]\n",
      " [ 1 33]]\n"
     ]
    }
   ],
   "source": [
    "shoe_sales = shoe_sales.T\n",
    "print \"Shape:\",shoe_sales.shape\n",
    "print \"Data:\", \"\\n\", shoe_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This is the approach we are going to take\n",
    "#We are going to randomly shuffle the labels. Then compute the mean between the two groups. \n",
    "#Find the % of times when the difference between the means computed is greater than what we observed above\n",
    "#If the % of times is less than 5%, we would make the call that the improvements are real"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.shuffle(shoe_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1, 29],\n",
       "       [ 1, 24],\n",
       "       [ 1, 19],\n",
       "       [ 1, 16],\n",
       "       [ 1, 28],\n",
       "       [ 1, 41],\n",
       "       [ 1, 27],\n",
       "       [ 0, 18],\n",
       "       [ 1, 33],\n",
       "       [ 0, 24],\n",
       "       [ 0, 21],\n",
       "       [ 1, 32],\n",
       "       [ 1, 31],\n",
       "       [ 0, 23],\n",
       "       [ 0, 19],\n",
       "       [ 0, 17],\n",
       "       [ 0, 27],\n",
       "       [ 0, 21],\n",
       "       [ 1, 32],\n",
       "       [ 0, 23],\n",
       "       [ 0, 24],\n",
       "       [ 0, 33],\n",
       "       [ 0, 35],\n",
       "       [ 1, 23]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shoe_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "experiment_label = np.random.randint(0,2,shoe_sales.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0,\n",
       "       1])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0 29]\n",
      " [ 1 24]\n",
      " [ 1 19]\n",
      " [ 1 16]\n",
      " [ 1 28]\n",
      " [ 1 41]\n",
      " [ 0 27]\n",
      " [ 0 18]\n",
      " [ 0 33]\n",
      " [ 0 24]\n",
      " [ 0 21]\n",
      " [ 1 32]\n",
      " [ 0 31]\n",
      " [ 1 23]\n",
      " [ 0 19]\n",
      " [ 1 17]\n",
      " [ 0 27]\n",
      " [ 1 21]\n",
      " [ 1 32]\n",
      " [ 0 23]\n",
      " [ 1 24]\n",
      " [ 1 33]\n",
      " [ 0 35]\n",
      " [ 1 23]]\n"
     ]
    }
   ],
   "source": [
    "experiment_data = np.array([experiment_label, shoe_sales[:,1]])\n",
    "experiment_data = experiment_data.T\n",
    "print experiment_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "experiment_diff_mean =  experiment_data[experiment_data[:,0]==1].mean() \\\n",
    "                        - experiment_data[experiment_data[:,0]==0].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26223776223776341"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_diff_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Like the previous notebook, let's repeat this experiment 100 and then 100000 times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def shuffle_experiment(number_of_times):\n",
    "    experiment_diff_mean = np.empty([number_of_times,1])\n",
    "    for times in np.arange(number_of_times):\n",
    "        experiment_label = np.random.randint(0,2,shoe_sales.shape[0])\n",
    "        experiment_data = np.array([experiment_label, shoe_sales[:,1]]).T\n",
    "        experiment_diff_mean[times] =  experiment_data[experiment_data[:,0]==1].mean() \\\n",
    "                        - experiment_data[experiment_data[:,0]==0].mean()\n",
    "    return experiment_diff_mean    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "experiment_diff_mean = shuffle_experiment(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.83333333],\n",
       "       [ 0.7       ],\n",
       "       [-0.33333333],\n",
       "       [ 0.54444444],\n",
       "       [-1.0625    ],\n",
       "       [ 0.61428571],\n",
       "       [ 3.36713287],\n",
       "       [ 1.3       ],\n",
       "       [ 0.3       ],\n",
       "       [ 0.66666667]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_diff_mean[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10a6c56d0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAECCAYAAAAIMefLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEARJREFUeJzt3X+M5Hddx/Hn7l5vZndm7kpyK2psIAH5BKMoVIIgoW0K\nBqKmQuAPU4kgBBASiqINVFJjgkjE1kCsRBCoYsTQpkDQADVn0yYkNAcWFDHv/iAkog3uHaWdmdvd\n+zHjHztn7sruzo+dme++756PpMn8+H6/n1e/O/ea73xnvt/vQr/fR5KU02LVASRJk7PEJSkxS1yS\nErPEJSkxS1ySErPEJSmxA7s9WUq5DPg48DSgBrwX+C7wj8CDg8k+HBGfnmVISdL2di1x4HpgLSJe\nW0p5CvAN4I+AWyLi1pmnkyTtaliJ3wHcObi9CJwGrgRKKeU64CHgHRHRmV1ESdJOFkY5YrOU0gI+\nB3wEqAPfiIgHSik3AU+JiN+fbUxJ0naGfrFZSrkC+BfgbyPiH4DPRMQDg6c/Czx3hvkkSbsY9sXm\nU4G7gbdGxD2Dh79YSnl7RBwDrgW+OmyQfr/fX1hY2HNYSbqEjFSau+5OKaV8EHgNEOc9/C7gFrb2\njz8KvGmEfeL9tbX2KHn2pdXVFlnzZ84O5q+a+auzutoaqcR33RKPiBuAG7Z56sWThJIkTZcH+0hS\nYpa4JCVmiUtSYpa4JCVmiUtSYpa4JCVmiUtSYpa4JCVmiUtSYpa4JCVmiUtSYpa4JCVmiUtSYsMu\nz6aLUK/Xo9vtVh0DgEajweKi2xLSpCzxS1C32+XosUeo1ZcrzbG5sc61z38GrVar0hxSZpb4JapW\nX2Z5pVl1DEl75OdYSUrMEpekxCxxSUrMEpekxCxxSUrMEpekxCxxSUrMEpekxCxxSUrMEpekxCxx\nSUrMEpekxCxxSUrMEpekxCxxSUrM84nrkjfLKx3Van3a7c7I03ulI43LEtclb5ZXOmo26nS6GyNN\n65WONAlLXGJ2VzpaadQ52/efmWbHz22SlJglLkmJWeKSlNiuO+tKKZcBHweeBtSA9wL/CdwO9IBv\nAm+LiP5sY0qStjNsS/x6YC0iXgK8HLgNuAW4afDYAnDdbCNKknYyrMTvAG4+b9rTwPMi4r7BY18A\nXjqjbJKkIXbdnRIRXYBSSoutQn8P8GfnTdIBDs8snSRpV0N/wFpKuQK4C7gtIj5VSvnT855uAT8Y\nZaDV1dwHMGTO/+TstVqfZqPOSqNeUaItSwtnOHKkyaFDu6/bWa/7Wa+PVnO05Y66PuYt82sf8ucf\nZtgXm08F7gbeGhH3DB5+oJRyVUTcC7wCODrKQGtr7T0FrdLqaitt/u2yt9sdOt2Nyg9CWT+5wfHj\nHTY3F3acZh7rfpbro9Ws0+6MdsTmKOtj3jK/9iF3/lHffIa9am9ia3fJzaWUc/vGbwA+VEo5CHwL\nuHPSkJKkvRm2T/wGtkr7ya6eSRpJ0lg82EeSErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPE\nJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkx\nS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySErPEJSkxS1ySEjtQdQBdunq9\nHp1OZ9dparU+7fbu0+xVp9Oh3+vPdAxpVixxVebU5jr3ff0JDh2+fMdpmo06ne7GTHM8/tgJ6itN\nVmY6ijQblrgqVasvs7zS3PH5lUads/3Zvkw31rszXb40S+4Tl6TELHFJSmykz6mllBcA74+Ia0op\nzwU+Dzw0ePrDEfHpWQWUJO1saImXUm4EfgM49xOBK4FbI+LWWQaTJA03yu6Uh4FXAQuD+1cCv1xK\nubeU8tellJ2/lZIkzdTQEo+Iu4Az5z10P/B7EXEV8G3gD2eUTZI0xCRfbH4mIh4Y3P4s8Nwp5pEk\njWGSH+B+sZTy9og4BlwLfHWUmVZXWxMMtX9kzv/k7LVan2ajzkqjXlGiLafW6ywuLdFq7p5j2PPz\nyjGpUZe7tHCGI0eaHDq0v15rmV/7kD//MOOU+Lnjkt8C3FZKOQ08CrxplJnX1tpjRts/VldbafNv\nl73d7tDpbsz8IJphuic3WFg8wIGDOx+R2WrWaXdme8TmKDkmNU7+9ZMbHD/eYXNzYfjEc5L5tQ+5\n84/65jPSv+KI+A7wosHtbwAvnjSYJGl6PNhHkhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKz\nxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKzxCUpsWpPKC3p//V6PTqdzvAJ56DRaLC46DZeBpa4\ntE+c2lznvq8/waHDl1eaY3NjnWuf/wxarYv7ijgXC0tc2kdq9WWWV5pVx1Aifl6SpMQscUlKzBKX\npMQscUlKzBKXpMQscUlKzJ8YzlGv16Pb7c51zFqtT7t94QEknU6Hfq8/1xySZsMSn6Nut8vRY49Q\nqy/Pbcxmo06nu3HBY48/doL6SpOVuaWQNCuW+JzN+2COlUads/0L/8wb6/P9NCBpdtwnLkmJWeKS\nlJglLkmJWeKSlJglLkmJWeKSlJglLkmJWeKSlJglLkmJWeKSlJglLkmJjXTulFLKC4D3R8Q1pZRn\nArcDPeCbwNsiwlPiSVIFhm6Jl1JuBD4K1AYP3QrcFBEvARaA62YXT5K0m1F2pzwMvIqtwgZ4XkTc\nN7j9BeClswgmSRpuaIlHxF3AmfMeWjjvdgc4PO1QkqTRTPLFZu+82y3gB1PKIkka0yQXhXiglHJV\nRNwLvAI4OspMq6utCYbaP6aRv1br02zUWWnUp5BodK3mheOdWq+zuLT0Q4/P26g5Zp1z1utj1OXu\nl7/L0sIZjhxpcujQ1mvef7v72zglfu4XKO8EPlpKOQh8C7hzlJnX1tpjRts/VldbU8nfbnfodDd+\n6Eo7s9Rq1ml3Lrw8W/fkBguLBzhwcGOHueZjlBzb5a8ix6TGyb9f/i7rJzc4frzD5ubC1F77Vcmc\nf9Q3n5HaJCK+A7xocPsh4OoJc0mSpsiDfSQpMUtckhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtc\nkhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKzxCUpMUtckhKz\nxCUpMUtckhI7UHUASXqyXq9Ht9vd83JqtT7tdmdPy2g0Giwu7t/tXUtc0r7T7XY5euwRavXlPS2n\n2ajT6W5MPP/mxjrXPv8ZtFqtPeWYJUtc0r5Uqy+zvNLc0zJWGnXO9i/umtu/nxEkSUNZ4pKUmCUu\nSYlZ4pKUmCUuSYlZ4pKUmCUuSYlZ4pKUmCUuSYlZ4pKUmCUuSYlZ4pKU2MRnhiml/Cvw+ODutyPi\nDdOJJEka1UQlXkqpA0TENdONI0kax6Rb4j8LrJRSvjRYxk0Rcf/0YkmSRjFpiXeBD0TEx0opPwl8\noZTyrIjoTTGbpAr0ej06na2r4UzjyjiT6HQ69Hv9uY+b0aQl/iDwMEBEPFRKOQH8GPDfO82wurp/\nr4wximnkr9X6NBt1Vhr1KSQaXat54Xin1ussLi390OPzNmqOWeec9foYdbn75+/yOF978HscvvwU\nfPv7lWR47LE1VlZaU1kXe1nG0sIZjhxpcujQ/u2vSUv89cBzgLeVUn4cOAQ8utsMa2vtCYeq3upq\nayr52+0One7GXK800mrWaXcuvDxV9+QGC4sHOHBw8stWTcMoObbLX0WOSY2Tf7/9Xc72D8xl/W/n\n7NlFOt3NPa+LveZfP7nB8eMdNjcX9pRjEqNuOE7aJh8DPlFKuW9w//XuSpGk+ZuoxCPiDPDaKWeR\nJI3Jg30kKTFLXJISs8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJIS\ns8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJISs8QlKTFLXJISOzCPQb73v2ucONGZ\nx1A7qh2scehQq9IMkjRtcynx+7/1fdrdjXkMtaPWgU1e+LxnV5pBkqZtLiV+sFbj4On+PIba0SKn\nKx1fkmbBfeKSlJglLkmJWeKSlJglLkmJWeKSlJglLkmJzeUnhvtBr9ej3W5PNG+t1qfd3vvBSp1O\nh36v2p9aSrq4XDIlvrmxztFjj1CrL489b7NRpzOFg5Uef+wE9ZUmK3tekiRtuWRKHKBWX2Z5pTn2\nfCuNOmf7e19VG+vdPS9Dks7nPnFJSswSl6TEJtpHUEpZBP4SeA6wCbwxIh6ZZjBJ0nCTbon/GnAw\nIl4EvAu4ZXqRJEmjmrTEfxH4IkBE3A/8/NQSSZJGNmmJHwKeOO/+2cEuFknSHE36u7kngPMvk7MY\nEb2dJt7snGCzU+1FIS7rn2bz9PpE8y4tnGH95N7zb25ssLC4xPrJ+V3laLvsVeTYzig5prXu95pj\nUuPk349/l3ms/2EZ9mKv+Tc3JuuMeZq0xL8M/CpwRynlF4B/223i61525cKE40iSdjFpiX8GeFkp\n5cuD+6+fUh5J0hgW+n3P5SFJWfllpCQlZolLUmKWuCQlZolLUmIzPxVtKaUB/D1wOXAK+M2I+J9Z\njzstpZTDwN+x9bv4g8DvRsRXqk01mVLKK4FXR8T1VWcZ5mI5P08p5QXA+yPimqqzjKqUchnwceBp\nQA14b0R8vtpUoyulLAEfBZ4F9IG3RMR/VJtqfKWUHwG+BlwbEQ/uNN08tsTfCByLiKvYKsMb5zDm\nNP0O8M8RcTXwOuC2StNMqJTyQeB9QJbf7Kc/P08p5Ua2yqRWdZYxXQ+sRcRLgJcDf1FxnnH9CtCL\niBcD7wH+uOI8Yxu8kf4VMPQiBDMv8Yg4Vx6w9c7+2KzHnLI/Bz4yuH0ZsP8P4drel4HfJk+JXwzn\n53kYeBV51vk5dwA3D24vAmcqzDK2iPgc8ObB3aeTr3MAPgB8GHh02IRT3Z1SSnkD8I4nPfy6iPha\nKeUo8NPAL01zzGkakv9HgU8CN8w/2eh2+X/4dCnl6goiTWrb8/PsdnqH/SYi7iqlPL3qHOOKiC5A\nKaXFVqH/QbWJxhcRZ0sptwOvBF5dcZyxlFJex9YnobtLKe9myEbAXA/2KaUU4J8i4plzG3QKSik/\nA3wKeGdEfKnqPJMalPibI+LXq84yTCnlFuArEXHH4P5/RcQVFcca26DEPxURL6w6yzhKKVcAdwG3\nRcTtFceZWCnlqcD9wLMjIsWn6FLKvWzty+8DPwcEcF1EfG+76efxxea7ge9GxCfZ2r+T6qNZKeWn\n2NoaeU1E/HvVeS4hY52fR9MzKL67gbdGxD1V5xlXKeW1wE9ExJ+wtfuzN/gvhcH3hwCUUu5ha8Nr\n2wKH+Vwo+WPA35RSfgtYIt95Vt7H1q9SPrT1QYIfRMQrq400sXPv7hlcTOfnybLOz7kJOAzcXEo5\nt2/8FRFR7alIR3cncPtgi/Yy4IaI2Kw408x47hRJSsyDfSQpMUtckhKzxCUpMUtckhKzxCUpMUtc\nkhKzxCUpMUtckhL7P8BDb+El45uCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106906f50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(experiment_diff_mean, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data: Difference in mean greater than observed: []\n",
      "Number of times diff in mean greater than observed: 0\n",
      "% of times diff in mean greater than observed: 0.0\n"
     ]
    }
   ],
   "source": [
    "#Finding % of times difference of means is greater than observed\n",
    "print \"Data: Difference in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference]\n",
    "\n",
    "print \"Number of times diff in mean greater than observed:\", \\\n",
    "            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]\n",
    "print \"% of times diff in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Repeat the above for 100,000 runs and report the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Is the result by chance? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the justification for shuffling the labels? \n",
    "\n",
    ">Thought process is this: If price optimization had no real effect, then, the sales before optimization would often give more sales than sales after optimization. By shuffling, we are simulating the situation where that happens -  sales before optimization is greater than sales after optimization. If many such trials provide improvements, then, the price optimization has no effect. In statistical terms, *the observed difference could have occurred by chance*. \n",
    "\n",
    "Now, to show that the same difference in mean might lead to a different conclusion, let's try the same experiment with a different dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])\n",
    "after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean sales before price optimization: 237.5\n",
      "Mean sales after price optimization: 241.666666667\n",
      "Difference in mean sales: 4.16666666667\n"
     ]
    }
   ],
   "source": [
    "print \"Mean sales before price optimization:\", np.mean(before_opt)\n",
    "print \"Mean sales after price optimization:\", np.mean(after_opt)\n",
    "print \"Difference in mean sales:\", np.mean(after_opt) - np.mean(before_opt) #Same as above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),\n",
    "np.append(before_opt, after_opt)], dtype=int)\n",
    "shoe_sales = shoe_sales.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10b414050>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAECCAYAAAAW+Nd4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGShJREFUeJzt3W+MXNd53/Hv7FI7s7szS7/g2IpUI6nd6KmSlHYkE64l\ng6QgRbIEG0oDREYUuJKRSLXMCHIrIK23igAbVNTEkdoSdpiEdEIpKlzAhJPYJUTJJQIuvalCxpGU\nqHIem0qcwEbS7tIkd2Y4M8vdmb6Yu83szv/d2fl3fh+A0M5z7869RzP7u3fOOXNvrFwuIyIiYRnr\n9w6IiEjvKfxFRAKk8BcRCZDCX0QkQAp/EZEAKfxFRAK0o9UKZvYA8GD0cBJ4D/BB4L8CJeAN4IC7\nl83sIeBhYAU46O4nzGwSeAFIAxngAXdf7HZDRESkfbFO5vmb2eeB14CPAM+4+5yZHQZeAl4BXgZu\npnKQ+AbwPuCXgKS7f9bMPgp8wN0/1d1miIhIJ9ru9jGz9wE/5u5HgZvdfS5a9CJwB7AHmHf3q+6+\nBJwHdgO3AiejdU9G64qISB910uc/C3wm+jlWVc8AO4EZ4HKD+tKGmoiI9FFb4W9mbwNucPfTUalU\ntXgGuEQl4FNV9VSd+lpNRET6qOWAb2QvcKrq8atmti86GNwdLTsLPGVmcSAB3EhlMHgeuAc4F607\nRxPlcrkci8WarSIiIrU6Cs52w/8G4K2qx48DR8xsAngTOB7N9jkEnKHyiWLW3YvRgPBzZnYGKAL3\nN937WIyFhUwnbRgq6XRK7RtSo9w2UPuGXTqdar1SlY5m+/RIedRfILVvOI1y20DtG3bpdKqjM399\nyUtEJEAKfxGRACn8RUQC1O6Ar4gApVKJXC5Xd9n09DRjYzqfkuGg8BfpQC6X49S5t4gnJtfVi4U8\nt+95N6lUZzMuRPpF4S9SR70z/Hi8TDabZWIiweRUsk97JtIdCn+ROuqd4SenE3z/e98nMZVkqo/7\nJtINCn+RBuKJyXVn+FPTCeKJRB/3SKR7NDolIhIghb+ISIAU/iIiAVL4i4gESOEvIhIghb+ISIAU\n/iIiAVL4i4gESOEvIhIghb+ISIAU/iIiAVL4i4gESOEvIhIghb+ISIB0SWeRbaTbPsqgUvjLyOtn\nAOu2jzKoWoa/mX0a+AhwDfB5YB44BpSAN4AD7l42s4eAh4EV4KC7nzCzSeAFIA1kgAfcfXE7GiLS\nSL8DeONNYUQGQdNTHjPbD3zA3W8B9gPvAp4BZt19LxAD7jWza4FHgVuAu4CnzWwCeAR4PVr3eeCJ\nbWqHSFNrAVz9b+PBQCQkrT7v3gn8pZn9IfA14KvAze4+Fy1/EbgD2APMu/tVd18CzgO7gVuBk9G6\nJ6N1RUSkz1p1+6SBdwIfpnLW/zUqZ/trMsBOYAa43KC+tKEmIiJ91ir8F4FvufsK8G0zKwDXVy2f\nAS5RCfjqjtNUnfparaV0erQHwdS+3orHyySnE0xNr7/5+nhshV27kszM1O5vo9+ZnkowNj5OKtne\nc21m2/00aK9dt416+zrRKvy/ATwGPGtm1wFTwCkz2+fup4G7gVPAWeApM4sDCeBGKoPB88A9wLlo\n3bnaTdRaWMhsoinDIZ1OqX09lslkyeYKrJbXv93zVwosLmYpFmNt/U4qmSB3pUBsbAc7JgptPddm\ntt0vg/jadVMI7etE0/CPZuzsNbOzVMYHPgl8FzgSDei+CRyPZvscAs5E6826e9HMDgPPmdkZoAjc\n32mDRESk+1pO9XT3f1+nvL/OekeBoxtqeeC+ze6ciIhsD33JS6QPSqUS2Wy24XJ9+1e2m8JfpA+W\ni3nmXltiZufbapbp27/SCwp/kT7RN3+ln/S5UkQkQAp/EZEAKfxFRAKk8BcRCZDCX0QkQAp/EZEA\naaqnjIxGd+zKZrOUS+U+7JHI4FL4y8hodMeuyxcvkJhKMtWn/RIZRAp/GSn1vjhVyNe/f69IyNTn\nLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/iEiAFP4iIgFS\n+IuIBKitC7uZ2Z8Dl6OHfw08DRwDSsAbwAF3L5vZQ8DDwApw0N1PmNkk8AKQBjLAA+6+2NVWiIhI\nR1qGv5klANz9tqraV4FZd58zs8PAvWb2CvAocDMwCXzDzL4OPAK87u6fNbOPAk8An+p+U0Q6UyqV\nyGazdZfpHgAy6to5838PMGVmL0Xr/0fgJnefi5a/CNwJrALz7n4VuGpm54HdwK3Ar0XrngR+pYv7\nL7Jpy8U8c68tMbPzbTXLdA8AGXXthH8O+Jy7f9HMfpRKgFfLADuBGf6xa2hjfWlDTWQg1Lv+P+ge\nADL62gn/bwPnAdz9O2Z2AfjJquUzwCUqAZ+qqqfq1NdqTaXTqVarDDW1b3vE42WS0wmmphPr6sv5\nBGPj46SS7dWbLZueql8fj62wa1eSmZn1be90n5o9Vy/ovRmOdsL/41S6bw6Y2XVUAvxlM9vn7qeB\nu4FTwFngKTOLAwngRiqDwfPAPcC5aN252k2st7CQ2URThkM6nVL7tkkmkyWbK7BaXv+2zl0pEBvb\nwY6JQlv1RstSyUTD38lfKbC4mKVYjG1pn5o913bTe3O4dXpgayf8vwj8npmthfbHgQvAETObAN4E\njkezfQ4BZ6hMIZ1192I0IPycmZ0BisD9He2hBKnRzdjXTE9PMzammcoim9Uy/N19BfhYnUX766x7\nFDi6oZYH7tvk/kmgGt2MHaBYyHP7nneTSukjvMhm6QbuMrAaDcYOokbTRjVlVAaVwl+kCxpNG9WU\nURlUCn+RLqn3SUVTRmVQacRMRCRAOvMXGRLNZkBp9pN0SuEvMiQazYDS7CfZDIW/yBAZphlQMtj0\nOVFEJEAKfxGRACn8RUQCpPAXEQmQwl9EJEAKfxGRACn8RUQCpPAXEQmQwl9EJEAKfxGRAOnyDjJ0\ndOMUka1T+MvQ0Y1TRLZO4S9DaZRvnKJPNtILCn+RAaNPNtILCn+RATTKn2xkMGi2j4hIgBT+IiIB\naqvbx8zeDnwTuB0oAcei/74BHHD3spk9BDwMrAAH3f2EmU0CLwBpIAM84O6LXW+FiIh0pOWZv5ld\nA/w2kANiwLPArLvvjR7fa2bXAo8CtwB3AU+b2QTwCPB6tO7zwBPb0goREelIO90+nwMOA38fPb7J\n3eein18E7gD2APPuftXdl4DzwG7gVuBktO7JaF0REemzpuFvZg8CC+7+clSKRf/WZICdwAxwuUF9\naUNNRET6rFWf/8eBspndAbwXeI5K//2aGeASlYBPVdVTdeprtZbS6VTrlYaY2tdaPF4mOZ1gajpR\ns2w5n2BsfJxUMrEt9WbLpqf6t+1G9fHYCrt2JZmZ2fr/d703w9E0/N1939rPZvbHwCeAz5nZPnc/\nDdwNnALOAk+ZWRxIADdSGQyeB+4BzkXrztGGhYVM5y0ZEul0Su1rQyaTJZsrsFqufYvmrhSIje1g\nx0RhW+qNlqWSib5tu1k9f6XA4mKWYjHGVui9Odw6PbB1+iWvMvA4cCQa0H0TOB7N9jkEnKHSlTTr\n7kUzOww8Z2ZngCJwf4fbE5EWGl0OAmB6epqxMc3ollpth7+731b1cH+d5UeBoxtqeeC+ze6ciLTW\n6HIQxUKe2/e8m1RKXR1SS5d3EBkB9S4HIdKMwl/6qlQqkcvVXrNGV7AU2V4Kf+mrXC7HqXNvEU9M\nrqvrCpYi20vhL32nK1iK9J6mAYiIBEjhLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/iEiAFP4i\nIgFS+IuIBEjhLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/\niEiAFP4iIgFqeQN3MxsHjgA3AGXgE0AROAaUgDeAA+5eNrOHgIeBFeCgu58ws0ngBSANZIAH3H1x\nG9oiIiJtaufM/8NAyd0/CDwB/CrwDDDr7nuBGHCvmV0LPArcAtwFPG1mE8AjwOvRus9HzyEiIn3U\nMvzd/Y+AfxM9/BHgInCzu89FtReBO4A9wLy7X3X3JeA8sBu4FTgZrXsyWldERPqoZbcPgLuvmtkx\n4KeBnwV+qmpxBtgJzACXG9SXNtQkMKVSiVwuV1PPZrOUS+U+7JFI2NoKfwB3f9DM3gGcBRJVi2aA\nS1QCPlVVT9Wpr9WaSqdTrVYZaiG2b2lpiVPnvkciMbWufvHiAlNTKVLJxLr6cj7B2Ph4Tb3Zsm7V\nmy2bnurftjutx8rLxONl4vH6B9dkMsnY2PoP/yG+N0PVzoDvx4B/4u5PA3lgFfgzM9vn7qeBu4FT\nVA4KT5lZnMrB4UYqg8HzwD3AuWjdudqtrLewkNlca4ZAOp0Ksn2ZTJaV1TFWy+vfcqurY2RzRXZM\nFNbVc1cKxMZ21NSbLetWvdGyVDLRt21vpn7xBxf5g1P/l5mdb6vZRrGQ5/Y97yaV+scwDPW9OSo6\nPbC1c+Z/HDhmZqeBa4DHgL8CjkQDum8Cx6PZPoeAM1TGEmbdvWhmh4HnzOwMlVlC93e0hyKyafHE\nJJNTyX7vhgygluHv7nngo3UW7a+z7lHgaJ3fv2+T+yci26BUKpHNZtfV4vEymUyW6enpmu4gGT1t\n9/mLtKNUKpHJ1H601sDuYFku5pl7bWldl1ByOsGFCxdruoNkNCn8pauy2Synzr1FPDG5rn754gUS\nU0mmGvye9N7GLqGp6QTZXO0YhIwmhb90Xb1+5kK+dpqniPSPOvZERAKk8BcRCZDCX0QkQAp/EZEA\nKfxFRAKk8BcRCZDCX0QkQAp/EZEAKfxFRAKk8BcRCZDCX0QkQAp/EZEAKfxFRAKk8BcRCZDCX0Qk\nQAp/EZEA6WYu0rFSqUQuV//mLMvLZd2uUWQIKPylY7lcru6tGgGuLmcolSd0u8YhVe/G7mt0Y/fR\novCXTal3q0aA8dgqVworfdgj6YZ6N3YHKBbyurH7iFH4i8g6jQ7sMlr0GU5EJEBNz/zN7Brgd4Ef\nBuLAQeBbwDGgBLwBHHD3spk9BDwMrAAH3f2EmU0CLwBpIAM84O6L29QWERFpU6sz/58HFtx9L/Ah\n4AvAM8BsVIsB95rZtcCjwC3AXcDTZjYBPAK8Hq37PPDE9jRDREQ60Sr8vww8WbXuVeAmd5+Lai8C\ndwB7gHl3v+ruS8B5YDdwK3AyWvdktK6IiPRZ024fd88BmFmKyoHgCeA3qlbJADuBGeByg/rShpqI\niPRZy9k+ZvZO4CvAF9z9S2b261WLZ4BLVAK+eg5Yqk59rdZSOj3a08mGvX3xeJnkdIKp6UTNsgv5\nyySn46SS65ct5xOMjY9vud7N59rMNqan+rftfrZvPLbCrl1JZmaG+7077H973dRqwPcdwMvAJ939\nj6Pyq2a2z91PA3cDp4CzwFNmFgcSwI1UBoPngXuAc9G6c7RhYSGziaYMh3Q6NfTty2SyZHMFVsv1\n3z7ZXJEdE4V1tdyVArGxHVuud/O5Ot1GKpno27b73b78lQKLi1mKxVjNcw2LUfjba6bTA1urM/9Z\nKl01T5rZWt//Y8ChaED3TeB4NNvnEHCGytjArLsXzeww8JyZnQGKwP0d7Z2IiGyLVn3+j1EJ+432\n11n3KHB0Qy0P3LeF/RMRkW2gL3mJiARI4S8iEiCFv4hIgBT+IiIBUviLiARI4S8iEiCFv4hIgBT+\nIiIBUviLiARIt3GUhkqlErlcrqaezWYpl8p92CPpF93YffQo/KWhXC7HqXNvEU9MrqtfvniBxFSS\nqT7tl/Sebuw+ehT+0lS9m3kX8rWfBmT06cbuo0Wf1UREAqTwFxEJkMJfRCRACn8RkQAp/EVEAqTw\nFxEJkMJfRCRACn8RkQDpS16iyziIBEjhL7qMg2xas2v+gK77M8gU/gLoMg6yOY2u+QO67s+gU/iL\nyJbomj/Dqa3wN7P3A//J3W8zs38GHANKwBvAAXcvm9lDwMPACnDQ3U+Y2STwApAGMsAD7r64De0Q\nEZEOtOyMM7NfBo4A8aj0LDDr7nuBGHCvmV0LPArcAtwFPG1mE8AjwOvRus8DT3S/CSIyiNbGAzKZ\nTM2/UqnU790LXjtn/ueBnwF+P3p8k7vPRT+/CNwJrALz7n4VuGpm54HdwK3Ar0XrngR+pVs7LiKD\nTfcAGGwtw9/dv2JmP1JVilX9nAF2AjPA5Qb1pQ016RNN6ZRe03jA4NrMgG/157UZ4BKVgK8+jKfq\n1NdqLaXTo31G0K/2LS0tcerc90gk1k/evHhxgampFKlkYl19OZ9gbHy87TrAhfxlktPxLT9Xs210\n67k2s43pqf5te5ja12zZeGyFXbuSzMz0/u9g1LOlE5sJ/1fNbJ+7nwbuBk4BZ4GnzCwOJIAbqQwG\nzwP3AOeidefqP+V6CwuZTezWcEinU31rXyaTZWV1jNXy+pd9dXWMbK7IjonCunruSoHY2I6262u6\n8VzNttGt5+p0G6lkom/bHrb2NVuWv1JgcTFLsRir+Z3t1M+/vV7o9MDWybcv1voFHgc+Y2Z/QuXg\ncdzd/w9wCDhD5WAw6+5F4DDw42Z2BvhF4DMd7Z2IiGyLts783f27VGby4O7fAfbXWecocHRDLQ/c\nt9WdFBGR7tL3rkVEAqTwFxEJkMJfRCRACn8RkQAp/EVEAqTwFxEJkMJfRCRACn8RkQDpZi4i0lPN\nbv2o2z72jsJ/BOnqnTLIdKnnwaDwH0G6IbsMOl3quf8U/iNKN2QXkWbUuSYiEiCd+Q+pRv36oL59\nGU4aCO4thf+QatSvD+rbl+GkgeDeUvgPsUaDZurbl2GlgeDe0ecoEZEA6cx/wGnOvohsB4X/gNOc\nfQmdBoK3h8J/CGjOvoRMA8HbQ+EvIgNPA8Hdp89LIiIB0pn/gNDArkhnmo0FgMYDWtn28DezMeA3\ngd1AEfhFd39ru7c7bDSwK9KZRmMBoPGAdvTizP+ngQl3v8XM3g88E9WCVCqVyGQyNfVsNsvEREID\nuyIdaDQWUO9TQTxeJpPJ6hNBpBfhfytwEsDd/9TM3teDbfZVs+vuLC8v8T//9C0SU+vP5XWGL9I9\n9T4VJKcTLCxc4AM//kMkk7UHjNAOCr0I/xlgqerxqpmNuXupB9vuikZhXipVmrDxDZPNZnnlf/9D\nTcADXF3OEBuf0Bm+yDbb+KlgajpBbPECc6/9XU1XUf5KLriDQi/Cfwmo7njravAvLy/z2uuv1V12\n/XXXMTOzc8vbyGaznH71b4jHE+vqly//gLHYOKkN27h8+QdMTqZINDiPLxby5K9kN9QKxMbGt1zv\n5nNtZttXl69QLKwOdfsaLRuPrfRt28PWvl60Yyvt22i5WODrr/xVzd9ysVhg30/+07oHhV7ZrnGL\nXoT/PPAR4Mtm9i+Bv2ixfiyd7qyx119/+yZ3rX27d9+w7duQYfAT/d6Bbab2haIX4f8HwE+Z2Xz0\n+OM92KaIiDQRK5c1h1xEJDSjN4ohIiItKfxFRAKk8BcRCZDCX0QkQANxYTczGweeBW4GJoAn3f1k\nNDX0vwArwMvu/tk+7uaWmdk/B14B3u7uy6PSPjPbCbxA5fscE8C/c/dXRqh9I3d9KjO7Bvhd4IeB\nOHAQ+BZwDCgBbwAH3H2oZ4SY2duBbwK3U2nXMUakfWb2aSrT6K8BPk9lWv0x2mzfoJz5fwzY4e4f\npHLdnxuj+m8BPxfV329m7+3XDm6Vmc1Qua5Roap8mNFo378Fvu7u+4EHgS9E9VF5/f7/9amA/0Dl\ndRx2Pw8suPte4ENUXrNngNmoFgPu7eP+bVl0gPttIEelPc8yIu0zs/3AB6L35H7gXXT4+g1K+N8J\nfN/M/gdwBPijKCwn3P1vonVeAu7o1w5uhZnFqLwJPw3ko9oMEB+F9gH/Gfid6OdrgLyZpRiR148N\n16cCRuH6VF8Gnox+HgOuAje5+1xUe5Hhfb3WfI7KCdbfR49HqX13An9pZn8IfA34KnBzJ+3rebeP\nmf0C8KkN5QUg7+4fNrO9wO8B97P+mkAZKke3gdagfX8L/Hd3/wszg8pReeM1j4a5fQ+6+zfN7Frg\n94HHgJ0MYfsaGPrrU23k7jmA6CD9ZeAJ4DeqVslSeQ2Hkpk9SOWTzctR90gs+rdmqNsHpIF3Ah+m\n8nf1NTpsX8/D392/CHyxumZmXwJORMvnzOwGaq8JNANc6tV+blaD9n0H+IUoOK+lchb8EUakfQBm\n9i+ALwGPu/uZ6JPN0LWvgW29PlW/mNk7ga8AX3D3L5nZr1ctTjG8rxdUriRQNrM7gPcCz1EJzDXD\n3r5F4FvuvgJ828wKwPVVy1u2b1C6fb4B3ANgZu8B/tbdM8Cymb0r6ja5E5hr8hwDy91/1N1vc/fb\ngH8A7hyl9pnZj1E5e/w5d38JwN2XGJH2URlIW3t/tnN9qoFnZu8AXgZ+2d2PReVXzWxf9PPdDO/r\nhbvvc/f90d/ca8C/Bk6OSvuoZOaHAMzsOmAKONVJ+wZitg+Vfv7DZva/osefqPrvfwPGgZfc/Vw/\ndq7LqkffR6V9v0plls+hqFvrkrv/K0anfaN4fapZKt0CT5rZWt//Y1RewwngTeB4v3ZuG5SBx4Ej\no9A+dz9hZnvN7CyVk/hPAt+lg/bp2j4iIgEalG4fERHpIYW/iEiAFP4iIgFS+IuIBEjhLyISIIW/\niEiAFP4iIgFS+IuIBOj/AVMVjN48eqv2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b47c490>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment_diff_mean = shuffle_experiment(100000)\n",
    "sns.distplot(experiment_diff_mean, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of times diff in mean greater than observed: 40473\n",
      "% of times diff in mean greater than observed: 40.473\n"
     ]
    }
   ],
   "source": [
    "#Finding % of times difference of means is greater than observed\n",
    "print \"Number of times diff in mean greater than observed:\", \\\n",
    "            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]\n",
    "print \"% of times diff in mean greater than observed:\", \\\n",
    "        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Did the conclusion change now? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effect Size\n",
    "\n",
    "> **Because you can't argue with all the fools in the world. It's easier to let them have their way, then trick them when they're not paying attention**  - Christopher Paolini\n",
    "\n",
    "In the first case, how much did the price optimization increase the sales on average?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The % increase of sales in the first case: 17.5438596491 %\n"
     ]
    }
   ],
   "source": [
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])\n",
    "\n",
    "print \"The % increase of sales in the first case:\", \\\n",
    "(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,\"%\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The % increase of sales in the second case: 1.75438596491 %\n"
     ]
    }
   ],
   "source": [
    "before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])\n",
    "after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])\n",
    "\n",
    "print \"The % increase of sales in the second case:\", \\\n",
    "(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,\"%\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Would business feel comfortable spending millions of dollars if the increase is going to be just 1.75%. Does it make sense? Maybe yes - if margins are thin and any increase is considered good. But if the returns from the price optimization module does not let the company break even, it makes no sense to take that path.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Someone tells you the result is statistically significant. The first question you should ask?\n",
    "\n",
    "# How large is the effect?\n",
    "\n",
    "To answer such a question, we will make use of the concept **confidence interval**\n",
    "\n",
    "In plain english, *confidence interval* is the range of values the measurement metric is going to take. \n",
    "\n",
    "An example would be: 90% of the times, the increase in average sales (before and after price optimization) would be within the bucket `3.4 and 6.7` (These numbers are illustrative. We will derive those numbers below)\n",
    "\n",
    "What is the *hacker's way* of doing it? We will do the following steps:\n",
    "\n",
    "1. From actual sales data, we sample the data with repetition (separately for before and after) - sample size will be the same as the original\n",
    "2. Find the differences between the mean of the two samples.\n",
    "3. Repeat steps 1 and 2 , say 100,000 times.\n",
    "4. Sort the differences. For getting 90% interval, take the 5% and 95% number. That range gives you the 90% confidence interval on the mean.\n",
    "5. This process of generating the samples is called **bootstrapping**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load the data\n",
    "before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])\n",
    "after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#generate a uniform random sample\n",
    "random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sample before optimization: [23 21 19 24 35 17 18 24 33 27 21 23]\n",
      "Bootstrapped sample before optimization:  [21 17 19 21 33 27 24 18 18 19 24 24]\n"
     ]
    }
   ],
   "source": [
    "print \"Actual sample before optimization:\", before_opt\n",
    "print \"Bootstrapped sample before optimization: \", random_before_opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean for actual sample: 23.75\n",
      "Mean for bootstrapped sample: 22.0833333333\n"
     ]
    }
   ],
   "source": [
    "print \"Mean for actual sample:\", np.mean(before_opt)\n",
    "print \"Mean for bootstrapped sample:\", np.mean(random_before_opt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual sample after optimization: [31 28 19 24 32 27 16 41 23 32 29 33]\n",
      "Bootstrapped sample after optimization:  [33 41 27 32 28 41 33 41 41 31 29 19]\n",
      "Mean for actual sample: 27.9166666667\n",
      "Mean for bootstrapped sample: 33.0\n"
     ]
    }
   ],
   "source": [
    "random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)\n",
    "print \"Actual sample after optimization:\", after_opt\n",
    "print \"Bootstrapped sample after optimization: \", random_after_opt\n",
    "print \"Mean for actual sample:\", np.mean(after_opt)\n",
    "print \"Mean for bootstrapped sample:\", np.mean(random_after_opt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference in means of actual samples: 4.16666666667\n",
      "Difference in means of bootstrapped samples: 10.9166666667\n"
     ]
    }
   ],
   "source": [
    "print \"Difference in means of actual samples:\", np.mean(after_opt) - np.mean(before_opt)\n",
    "print \"Difference in means of bootstrapped samples:\", np.mean(random_after_opt) - np.mean(random_before_opt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Like always, we will repeat this experiment 100,000 times. \n",
    "\n",
    "def bootstrap_experiment(number_of_times):\n",
    "    mean_difference = np.empty([number_of_times,1])\n",
    "    for times in np.arange(number_of_times):\n",
    "        random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)\n",
    "        random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)\n",
    "        mean_difference[times] = np.mean(random_after_opt) - np.mean(random_before_opt)\n",
    "    return mean_difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10b5a60d0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAECCAYAAAAW+Nd4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGhdJREFUeJzt3X+M2/d93/En785H3vFIuYi4OGmDbnHhd90ASmJXaCpn\nkj171qwl1RoMHuA0cIxEnl3B8IBsGXZTPbiQo7SpjUZoJgxWGtlQUKAxkgyFYMWDVujU6w8rqa3G\nc/K25aQItnbpnWbfkRTJ04ncH/zyRFHUkbwjj0d+Xg/gIPLNz5Gfj/i91/fL748PY5VKBRERCctI\nvzsgIiIbT+EvIhIghb+ISIAU/iIiAVL4i4gESOEvIhKgsdUeNLMR4ChwC1AG9gGXgWPR/deA/e5e\nMbN9wMPAMnDQ3U+Y2QRwHMgAWeBBd5/v0VhERKRNrbb87wWS7v5R4LeBLwBPA9PuvhOIAXvN7Cbg\nMWAHsBs4ZGbjwKPAuajt88CB3gxDREQ60Sr8C8AWM4sBW4Al4HZ3n4kefxG4B9gOzLr7JXdfBM4D\n24A7gJNR25NRWxER6bNVd/sAs0AC+CHwLuDjwM66x7NUVwppYOE69cWGmoiI9FmrLf/PU92iN+BD\nVHfd3FD3eBp4h2rAp+rqqSb1Wk1ERPqs1ZZ/kitb7m9H7V8xs13ufhq4DzgFvAw8ZWZxqp8UbqV6\nMHgW2AOcjdrO0EKlUqnEYrE1DEVEJGgdBWdstYndzOxG4GvAVqpb/L8PfA94FhgHXgf2RWf7fJbq\n2T4jwFPu/q3obJ/ngPcAJeABd/+HFn2qzM1lOxnDQMlkUmh8g2mYxwYa36DLZFLdC/8+UfgPsGEe\n3zCPDTS+Qddp+OsiLxGRACn8RUQCpPAXEQmQwl9EJEAKfxGRACn8RUQC1OoiL5GhVy6Xyefz19ST\nySQjI9o+kuGk8Jfg5fN5Tp19i3hiYqVWKha4e/vNpFKpVX5TZHAp/EWAeGKCicmpfndDZMPoM62I\nSIAU/iIiAVL4i4gESOEvIhIgHfCVoDQ7rTOXy1Epb7rZbUV6SuEvQWl2WufC2xdITE4x2cd+iWw0\nhb8Ep/G0zmLh2gu8RIad9vmLiARI4S8iEiCFv4hIgBT+IiIBannA18weBD4d3Z0APgh8FPgyUAZe\nA/a7e8XM9gEPA8vAQXc/YWYTwHEgA2SBB919vtsDERGR9rXc8nf359z9Lne/C/gu8BjwBDDt7juB\nGLDXzG6KHtsB7AYOmdk48ChwLmr7PHCgN0MREZF2tb3bx8x+Gfgldz8K3O7uM9FDLwL3ANuBWXe/\n5O6LwHlgG3AHcDJqezJqKyIifdTJPv9p4MnodqyungW2AGlg4Tr1xYaaiIj0UVvhb2Y3Are4++mo\nVK57OA28QzXg67/5ItWkXquJiEgftXuF707gVN39V8xsV7QyuC967GXgKTOLAwngVqoHg2eBPcDZ\nqO0MLWQyw/3tSRpf/8TjFaaSCSaTiZXaUiHByOgoqakrtdHYMlu3TpFOXz2WzTy2btD4wtFu+N8C\nvFV3/3PAs9EB3deBF6KzfQ4DZ6h+oph295KZHQGeM7MzQAl4oNWLzc1lOxnDQMlkUhpfH2WzOXL5\nIpcrVxb9/MUisZExxsaLK7XCxSLz8zlKpSt7ODf72NZL4xtsna7Y2gp/d/+9hvtvAnc2aXcUONpQ\nKwD3d9QrERHpKV3kJSISIIW/iEiAFP4iIgFS+IuIBEjhLyISIH2Tl0gT5XKZXC53VS0er1AuVxgZ\n0TaTDD6Fvwyt9XxZ+1KpwMyri6S33LhSGxst85Ff+jlSKV0oJINP4S9Da71f1t74Xb+jseUe9FKk\nPxT+MtT0Ze0izWnnpYhIgBT+IiIBUviLiARI4S8iEiCFv4hIgBT+IiIBUviLiARI5/mLtKnZlA8A\nyWRSUz7IwFH4i7SpVCww8+o/XDXlQ6lY4O7tN2vKBxk4Cn+RDjReMSwyqPRZVUQkQC23/M3sPwEf\nB24A/gCYBY4BZeA1YL+7V8xsH/AwsAwcdPcTZjYBHAcyQBZ40N3nezEQERFp36pb/mZ2J/Cr7r4D\nuBN4P/A0MO3uO4EYsNfMbgIeA3YAu4FDZjYOPAqci9o+Dxzo0ThERKQDrbb87wW+b2bfBtLAfwA+\n4+4z0eMvRm0uA7Pufgm4ZGbngW3AHcDvRG1PAr/V5f6LrGicv7/duftFQtQq/DPA+4CPUd3q/xOq\nW/s1WWAL1RXDwnXqiw01kZ5onL+/k7n7RULTKvzngR+4+zLwhpkVgZ+tezwNvEM14OvPdUs1qddq\nLWUyw33anMbXG/F4hXe962eYTFbPxhmNXWZkdJTUVGKlzVIhsebahcICU8n4VbXR2DJbt06RTg/H\ne6plMxytwv/PgMeBZ8zsvcAkcMrMdrn7aeA+4BTwMvCUmcWBBHAr1YPBs8Ae4GzUdubal7jW3Fx2\nDUMZDJlMSuPrkWw2Ry5f5HKluljnLxaJjYwxNl5cabOeGkAuX7qqVrhYZH4+R6kUY9Bp2Rxsna7Y\nVg3/6IydnWb2MtWDw78J/C3wbHRA93Xghehsn8PAmajdtLuXzOwI8JyZnQFKwAOdDkhERLqv5ame\n7v4fm5TvbNLuKHC0oVYA7l9r50REpDd0ha9seo1n8dRshjl1ms33sxn6JdKKwl82vcazeGDzzKmz\nVCow8+riynw/m6VfIq0o/GUgbOY5dTZz30SuR59NRUQCpPAXEQmQdvuIdJG+8EUGhcJfpIsaDwCD\nDgLL5qTwl4HUbAt7s0zkpgPAMggU/jKQmm1hayI3kfYp/GVgNW5hFwvXXggmIs3pCJSISIAU/iIi\nAVL4i4gESOEvIhIghb+ISIAU/iIiAVL4i4gESOEvIhIghb+ISIAU/iIiAWpregcz+2tgIbr7I+AQ\ncAwoA68B+929Ymb7gIeBZeCgu58wswngOJABssCD7j7f1VGIiEhHWm75m1kCwN3vin4+AzwDTLv7\nTiAG7DWzm4DHgB3AbuCQmY0DjwLnorbPAwd6MxQREWlXO1v+HwQmzew7Ufv/DNzm7jPR4y8C9wKX\ngVl3vwRcMrPzwDbgDuB3orYngd/qYv9FRGQN2tnnnwe+5O67gUeArzc8ngW2AGmu7BpqrC821ERE\npI/a2fJ/AzgP4O5vmtkF4MN1j6eBd6gGfP1XFaWa1Gu1VWUyw/2NRxpfZ+LxClPJBJPJxEptqZBg\nZHSU1NT1a+206aR2obDAVDLe8WuOxpbZunWKdHrzv+9aNsPRTvg/RHX3zX4zey/VAH/JzHa5+2ng\nPuAU8DLwlJnFgQRwK9WDwbPAHuBs1Hbm2pe42txcdg1DGQyZTErj61A2myOXL3K5cmVxzV8sEhsZ\nY2y8eN1aO206qQHk8qWOX7Nwscj8fI5SKbbe/4qe0rI52DpdsbUT/l8FvmZmtdB+CLgAPBsd0H0d\neCE62+cwcIbq7qRpdy+Z2RHgOTM7A5SABzrqoYiIdF3L8Hf3ZeBTTR66s0nbo8DRhloBuH+N/RMR\nkR7QRV4iIgFS+IuIBEjhLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISIIW/iEiA2prPX0TWrlwuk8vl\nrqknk0lGRrT9Jf2h8BfpsaVSgZlXF0lvuXGlVioWuHv7zaRSmmhM+kPhL7IB4okJJian+t0NkRX6\nzCkiEiCFv4hIgBT+IiIBUviLiARI4S8iEiCFv4hIgBT+IiIBUviLiARIF3nJplIul8nn81fVcrkc\nlXKlTz0SGU5thb+Z/SPge8DdQBk4Fv37GrDf3Stmtg94GFgGDrr7CTObAI4DGSALPOju810fhQyN\nfD7PqbNvEU9MrNQW3r5AYnKKyT72S2TYtNztY2Y3AP8NyAMx4Blg2t13Rvf3mtlNwGPADmA3cMjM\nxoFHgXNR2+eBAz0ZhQyV2lQItZ94ItHvLokMnXb2+X8JOAL8fXT/NnefiW6/CNwDbAdm3f2Suy8C\n54FtwB3AyajtyaitiIj02arhb2afBubc/aWoFIt+arLAFiANLFynvthQExGRPmu1z/8hoGJm9wAf\nAp6juv++Jg28QzXg6+emTTWp12otZTLDPc2txnd98XiFqWSCyeSVXT1LhQQjo6OkpjqrrfX3rle7\nUFhgKhnvymuOxpbZunWKdHpzLQtaNsOxavi7+67abTP7U+AR4EtmtsvdTwP3AaeAl4GnzCwOJIBb\nqR4MngX2AGejtjO0YW4u2/lIBkQmk9L4VpHN5sjli1yuXFk08xeLxEbGGBsvdlRb6+9drwaQy5e6\n8pqFi0Xm53OUSvUfpPtLy+Zg63TF1ul5/hXgc8CTZvbnVFceL7j7T4HDwBmqK4Npdy9RPVbwATM7\nA3wWeLLD1xMRkR5o+zx/d7+r7u6dTR4/ChxtqBWA+9faORER6Q1d4SsiEiCFv4hIgBT+IiIBUviL\niARI4S8iEiCFv4hIgBT+IiIBUviLiARI4S8iEiCFv4hIgPQ1jiJ9UC6XyeVy19STySQjI9omk95T\n+Iv0wVKpwMyri6S33LhSKxUL3L39ZlIpTTssvafwF+mT2tdVivSDPl+KiARI4S8iEiDt9pG+KZfL\n5PP5q2q5XI5KudKnHomEQ+EvfZPP5zl19i3iiYmV2sLbF0hMTjHZx36JhEDhL33VeNCzWMiv0lpE\nukX7/EVEAqTwFxEJUMvdPmY2CjwL3AJUgEeAEnAMKAOvAfvdvWJm+4CHgWXgoLufMLMJ4DiQAbLA\ng+4+34OxiIhIm9rZ8v8YUHb3jwIHgC8ATwPT7r4TiAF7zewm4DFgB7AbOGRm48CjwLmo7fPRc4iI\nSB+1DH93/+/Av43u/mPgbeB2d5+Jai8C9wDbgVl3v+Tui8B5YBtwB3AyansyaisiIn3U1j5/d79s\nZseALwNfp7q1X5MFtgBpYOE69cWGmoiI9FHbp3q6+6fN7N3Ay0Ci7qE08A7VgK+fkSrVpF6rrSqT\nGe6JrTS+qni8wlQywWTyyuK0VEgwMjpKamr9tW4+F8CFwgJTyXjPXnM0tszWrVOk0/1bPrRshqOd\nA76fAn7O3Q8BBeAy8F0z2+Xup4H7gFNUVwpPmVmc6srhVqoHg2eBPcDZqO3Mta9ytbm57NpGMwAy\nmZTGF8lmc+TyRS5XriyG+YtFYiNjjI0X113r5nPV5PKlnr1m4WKR+fkcpVL9B+uNo2VzsHW6Ymtn\ny/8F4JiZnQZuAB4Hfgg8Gx3QfR14ITrb5zBwhurupGl3L5nZEeA5MztD9SyhBzrqoYiIdF3L8Hf3\nAvBvmjx0Z5O2R4GjTX7//jX2T0REekDTO4hsEvp2L9lICn+RTULf7iUbSeEvsono271ko+izpIhI\ngLTlLxtCX9wisrko/GVD6ItbRDYXhb9sGH1xi8jmoX3+IiIBUviLiARI4S8iEiCFv4hIgBT+IiIB\nUviLiARI4S8iEiCFv4hIgBT+IiIB0hW+0hONc/loHp+10Rz/0isKf+mJxrl8NI/P2miOf+kVhb/0\nTP1cPprHZ+00x7/0gj43iogEaNUtfzO7AfhD4OeBOHAQ+AFwDCgDrwH73b1iZvuAh4Fl4KC7nzCz\nCeA4kAGywIPuPt+jsYiISJtabfl/Ephz953AvwC+AjwNTEe1GLDXzG4CHgN2ALuBQ2Y2DjwKnIva\nPg8c6M0wRESkE63C/xvAE3VtLwG3uftMVHsRuAfYDsy6+yV3XwTOA9uAO4CTUduTUVsREemzVXf7\nuHsewMxSVFcEB4Dfq2uSBbYAaWDhOvXFhpqIrEOz0z916qd0quXZPmb2PuCbwFfc/Y/M7HfrHk4D\n71AN+PrzzlJN6rVaS5nMcJ/CFsL44vEKU8kEk8kEAEuFBCOjo6SmEivtel3r9vNfKCwwlYxv6Gs2\nry3wvTd+ypYblwAoFi/ysZ2/SDq9/uUqhGVTqlod8H038BLwm+7+p1H5FTPb5e6ngfuAU8DLwFNm\nFgcSwK1UDwbPAnuAs1HbGdowN5ddw1AGQyaTCmJ82WyOXL7I5Up1EctfLBIbGWNsvLjStte1bj8/\nQC5f2tDXXK1W+79dvjzC/HyOUinGeoSybA6rTldsrbb8p6nuqnnCzGr7/h8HDkcHdF8HXojO9jkM\nnKF6bGDa3UtmdgR4zszOACXggY56JyIiPdFqn//jVMO+0Z1N2h4FjjbUCsD96+ifiIj0gI4QiYgE\nSOEvIhIghb+ISIA0sZusW/30zfF4pXqmj6ZwFtnUFP6ybvXTN08lE+TyRU3hLLLJKfylK2rTDk8m\nE1yujGkKZ5FNTvv8RUQCpPAXEQmQwl9EJEAKfxGRACn8RUQCpPAXEQmQwl9EJEA6z186Un81b42u\n5hUZPAp/6Uj91bw1uppXZPAo/KVjtat5a3Q1r8jg0T5/EZEAKfxFRAKk8BcRCZDCX0QkQG0d8DWz\nXwG+6O53mdkvAMeAMvAasN/dK2a2D3gYWAYOuvsJM5sAjgMZIAs86O7zPRiHSLDK5TK5XO6aejKZ\nZGRE23fSXMslw8w+DzwLxKPSM8C0u+8EYsBeM7sJeAzYAewGDpnZOPAocC5q+zxwoPtDEAnbUqnA\nzKs/4c++/3crP6fOvnXN9Rgi9drZLDgPfIJq0APc5u4z0e0XgXuA7cCsu19y98Xod7YBdwAno7Yn\no7Yi0mW1029rP/XXYYg00zL83f2bVHfl1MTqbmeBLUAaWLhOfbGhJiIifbaWi7zKdbfTwDtUAz5V\nV081qddqLWUyqdaNBtggjy8erzCVTDCZTKzUlgoJRkZHSU1Va6mpxDW1xvsbUev2818oLDCVjG/o\na651nKOxZbZunSKd7mxZG+Rlsx3DPr5OrCX8XzGzXe5+GrgPOAW8DDxlZnEgAdxK9WDwLLAHOBu1\nnWn+lFebm8uuoVuDIZNJDfT4stkcuXyRy5Uri07+YpHYyBhj40VSUwmyueJVtcY2zX6vF7VuPz9A\nLl/a0Ndc6zgLF4vMz+coleo/qK9u0JfNVkIYXyc6ORWgNnPX54AnzezPqa48XnD3nwKHgTNUVwbT\n7l4CjgAfMLMzwGeBJzvqnYiI9ERbW/7u/rdUz+TB3d8E7mzS5ihwtKFWAO5fbyelPzSDp8jw0sRu\ncl2awXNw6dx/aUXhL6vSDJ6DqXru/yLpLTeu1ErFAndvv5lUSgc9ReEvMrQaV9wi9fT5T0QkQAp/\nEZEAKfxFRAKkff4C6LROkdAo/AXQaZ0ioVH4ywqd1ikSDoW/SCB04ZfUU/iLBEIXfkk9hX+AdHA3\nXLrwS2oU/gHSwV0RUfgHSgd3Ba4+DhCPV8hmq7d1HGD4KfxFAlZ/HGAqmSCXL+o4QCAU/gFo3Mev\n/ftSr/YpcDKZuOob2mS46Z0OQOM+fu3fFxGF/5C53pk84+OJlX382r8vq9H1AGFQ+A8Znckj69Xs\neoDCxTy/+oH3MDV15SQBrQwGW8/D38xGgP8KbANKwGfd/a1ev27IdCaPrFezZWjm1Z+srBB0UHjw\nbcSW/78Cxt19h5n9CvB0VJN10sVaspF0gdhw2YjwvwM4CeDuf2Vmv7wBrzl0rhf0f/m//i+JySs7\ndLSLRzaCjgsMvo0I/zSwWHf/spmNuHt5A15702sW6uVy9b+m/o9otaDXLh7ZaO0eF2i2LDergVYc\nG20jwn8RqN8xOHDBf+5vXmVpaWnlfowYN9/8fsbGxjt+rvqrKKEa6qdf+THxeGKltrDw/xiJjZJK\nb7mqNjGRItGwTV8qFihczNXdLxIbGV211k6btdZGY8sULhY39DU3apyXli5SKl7u2/9tr1/zeu/d\naq9Zb6lU5H/85Q+vWW6bLcuNtVKpyK4P/5OrVhzdlsno+ES9jQj/WeDjwDfM7CPA37RoH9tsb9I9\nd//Trj5fOp2+6v62bbd09flFpLnNli39tBHh/y3gn5vZbHT/oQ14TRERWUWsUtGZISIiodHRFRGR\nACn8RUQCpPAXEQmQwl9EJECbamI3M/t14F+7+yej+x8Bfh9YBl5y99/uZ/+6wcxiwP8G3ohKf+Hu\n033s0rqFMH+Tmf01sBDd/ZG7f6af/emGaLqVL7r7XWb2C8AxoAy8Bux394E+G6RhfB8G/gR4M3r4\niLv/cf96tz5mdgPwh8DPA3HgIPADOngPN034m9mXgXuBV+rKR4BPuPuPzeyEmX3I3V/tTw+75mbg\ne+7+a/3uSBcN9fxNZpYAcPe7+t2XbjGzzwO/AdSu5HoGmHb3GTM7AuwFvt2v/q1Xk/HdDjzj7s/0\nr1dd9Ulgzt0/ZWY/A5yjmp1tv4ebabfPLPAoEAMwszQQd/cfR49/B7inT33rptuBnzWz/xmt0Ibh\nCq+r5m8Chm3+pg8Ck2b2HTM7Fa3gBt154BNEf2/Abe4+E91+kcH/W2sc3+3AvzSz02Z21MwGfYa6\nbwBPRLdHgEt0+B5uePib2WfM7PsNP7c3+QjWOCdQFtjCAGk2VuDvgC+4+z8DvgAc728vu6Lp/E39\n6kwP5IEvuftu4BHg64M+Pnf/JtXdqTWxuts5BuxvrVGT8f0V8O/dfRfwI+C/9KVjXeLueXfPmVmK\n6orgAFfnecv3cMN3+7j7V4GvttG0cU6gNPBOTzrVI83GamYTRAulu8+a2Xv70bcuG/j5m1p4g+qW\nJO7+ppldAN4D/J++9qq76t+vFAP2t9aGb7l77ZjNt4HD/exMN5jZ+4BvAl9x9z8ys9+te7jle7hp\nt17cfRFYMrP3RwdJ7wVmWvzaIHgC+HcAZvZB4Cf97U5XzAJ7YOUgfav5mwbNQ1SPYxCtrNPA3/e1\nR933ipntim7fx3D8rdU7aWbbo9t3A9/tZ2fWy8zeDbwEfN7dj0Xljt7DTXPAN1KJfmoeAb4OjALf\ncfezfelVd30ROG5me6h+Avh0f7vTFcM+f9NXga+ZWe2P6aEh+mRT+3v7HPCsmY0DrwMv9K9LXVUb\n3yPAV8zsEtUV98P961JXTFPdrfOEmdX2/T8OHG73PdTcPiIiAdq0u31ERKR3FP4iIgFS+IuIBEjh\nLyISIIW/iEiAFP4iIgFS+IuIBEjhLyISoP8PoK6vAauRiewAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10bbb79d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean_difference = bootstrap_experiment(100000)\n",
    "sns.distplot(mean_difference, kde=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_difference = np.sort(mean_difference, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -6.66666667],\n",
       "       [ -6.33333333],\n",
       "       [ -6.08333333],\n",
       "       ..., \n",
       "       [ 13.16666667],\n",
       "       [ 13.16666667],\n",
       "       [ 15.        ]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_difference #Sorted difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.16666667,  8.08333333])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(mean_difference, [5,95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reiterating what this means: 90% of the times, the mean difference is between the limits as shown above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise: Find the 95% percentile for confidence intevals**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where do we go from here? \n",
    "\n",
    "First of all there are two points to be made.\n",
    "\n",
    "1. Whey do we need signficance testing if confidence intervals can provide us more information?\n",
    "2. How does it relate to the traditional statistical procedure of finding confidence intervals\n",
    "\n",
    "For the first one:\n",
    "\n",
    "What if sales in the first month after price changes was 80 and the month before price changes was 40. The difference is 40. And confidence interval,as explained above, using replacements, would always generate 40. But if we do the significance testing, as detailed above - where the labels are shuffled, the prices are equally likely to occur in both the groups. And so, significance testing would answer that there was no difference. But don't we all know that the data is **too small** to make meaningful inferences?\n",
    "\n",
    "For the second one:\n",
    "\n",
    "Traditional statistics derivation assumes normal distribution. But what if the underlying distribution isn't normal? Also, people relate to resampling much better :-) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
